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Using Multiple Measures to 
Predict Success in Students’ 
First College Math Course 


WestEd has undertaken a multiyear series of implementation 
studies intended to inform the California State University (CSU) 
system about the implementation of Executive Order 1110 (EO 
1110). A major policy adopted by the CSU Chancellor’s Office in 
2017, EO 1110 requires CSU campuses to eliminate noncredit 
developmental courses (often known as “remedial” courses) in 
Written Communication (WC) and Mathematics/Quantitative 
Reasoning (QR), change the process for how students are placed 
into WC and QR courses, and improve how students are supported 
to succeed. Earlier reports in the series have described the variation 
in course models and instructional approaches adopted by 
campuses, examined student progress during the first year of 
implementation, compared progress of students both before and 
after implementation of the new policy, and examined short-term 
outcomes for students who participated in revamped summer Early 
Start programs. This fifth and final report of the series focuses on 
how students are placed into entry-level math courses. Specifically, 
it focuses on the use of multiple measures for placing students and 
examines the predictive power of the measures for anticipating 
students’ success in their first college math course. 
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A significant aspect of EO 1110 is that it brought about a major change in how students 
entering the CSU system are placed into entry-level math and writing courses. Specifically, it 
required that CSU campuses discontinue using the English Placement Test (EPT) and Entry-Level 
Mathematics (ELM) placement exams and instead consider a series of “multiple measures” for 
placing students into entry-level QR and WC courses. In this report, WestEd analysts consider 
the extent to which the most commonly used measures are able to predict student success in 
entry-level math courses. 


Multiple Measures for Placement 


Previous researchers have found that factors such as high school grades tend to be much better 
predictors of student success in college-level math than a single placement test (Scott-Clayton, 
2012; Belfield & Crosta, 2012; Bostian, 2012; Burdman, 2012) and that multiple indicators used 
together tend to provide even greater predictive power. Aligned with that research, CSU’s new 
policy called for a move away from the use of placement exams and toward a new system that 
takes into account additional measures of college readiness. Rather than relying on a single 
high-stakes exam to determine whether a student entering the system was required to take 
noncredit developmental coursework, EO 1110 called for consideration of a collection of 
measures, including high school grades and test scores, to determine which type of 
baccalaureate-level courses the students should enroll in upon entry to the university. The 
move to multiple measures was particularly significant for QR placement, as campuses had 
relied heavily on the ELM exam to place students in math courses. In contrast, many campuses 
had already eliminated the use of the EPT and instead relied on a Directed Self Placement? 
process to place students in entry-level writing courses. Because the move to multiple 
measures had less impact in WC than in QR placement, this examination focuses on multiple 
measures identified for placement in QR courses. 


EO 1110 calls for the use of multiple measures to place students into one of four categories for 
QR (Box 1). Placement designations determine whether a student is considered to have already 
met their General Education requirement in QR and, if not, to determine whether they are 
ready to enroll directly in a traditional General Education math course or whether they would 
be better served by a course that provides additional supports in either a one- or two-semester 
format (Bracco et al., 2019). 


Most research on the efficacy of multiple measures for placement suggests that although high 
school grade point average (GPA) is the strongest predictor of success in entry-level 
coursework, combining that measure with other factors such as test scores, grades in specific 
courses, and potentially some noncognitive indicators can provide even greater predictive 


1 Directed Self Placement (DSP) is a process used on many campuses to allow students to determine, through self-reflection, 
the entry-level writing course that best fits their needs. Campuses design their own DSP tools, which typically include 
questions about educational goals, overall workload, and perceived skill level. 
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power (Ganga & Mazzariello, 2019). Researchers have identified different approaches to using 
multiple measures, some of which use the measures in combination and others that use a more 
hierarchical approach, relying on additional measures only if a student does not meet the 
threshold in one primary indicator, such as a test score (Bracco et al., 2014). 


Box 1 — California State University placement categories, based 
on multiple measures 


Category I: Has fulfilled the requirement for General Education (GE) Subarea A2 (for Written 
Communication, or WC) or B4 (for Mathematics/Quantitative Reasoning, or QR) 


e Student has met the CSU GE Breadth Subarea A2 and/or B4 requirement via Advanced 
Placement examination, International Baccalaureate examination, or transferable 
course 


Category II: Place in a GE Subarea A2 or B4 course 
e Student has met examination standards and/or multiple measures—informed standards 
Category III: Recommend placement in a supported GE Subarea A2 or B4 course 


e Based on new multiple measures, student needs additional academic support 


e Participation in the Early Start program is recommended and may be highly advisable 
for some students, particularly STEM majors 


Category IV: Require placement in a supported GE Subarea A2 or B4 course or the first term of 
an applicable stretch course 


e Based on new multiple measures, student needs additional academic support 
e Participation in the Early Start program is required 
Note: Placement categories for WC and QR courses are determined by a combination of student grades and test scores. Fora 


detailed description of the various ways in which a student can be placed into the different categories, see 
https://calstate.policystat.com/policy/6656541/latest/. 
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Research Questions 


In this study, WestEd analysts sought to address several related research questions: 


e What is the relationship between the measures used to determine placement and 
students’ success in their first college math course? 


e How does such a relationship differ by placement category and by student major (non- 
STEM versus STEM) within each category? 


e Is any measure more effective than other measures in predicting success in the first 
college math course? If so, how sensitive is that measure in predicting success? 


Study Sample 


Data from two student cohorts from all 23 CSU campuses were used in the study: students who 
entered the CSU system in fall 2018 as first-time, first-year students and those who entered in 
fall 2019 as first-time, first-year students. The sample was further refined to include students 
who had data on each of the four measures available for the largest percentage of students: 
the scale score on the SAT’s math section (SAT-math), the scale score on the Early Assessment 
Program/Smarter Balanced Assessment (EAP-math), weighted high school math GPA, and 
weighted high school overall GPA.” For the 2018 cohort, the first math course attempted was 
taken in fall 2018 or spring 2019. For the 2019 cohort, due to the uncertainties brought on by 
COVID-19 and the switch to remote learning in the middle of the spring 2020 term, only 
students who attempted a first math course in fall 2019 were included in the analyses. Tables 1 
and 2 list the study sample by placement category (at the beginning of fall semester) and by 
major within a category for each of the cohorts. The same analyses were done for both cohorts, 
and the outcomes were similar. Because the outcomes were so similar and because the 2019 
entering cohort included only one semester of data, this report focuses on the findings for the 
2018 entering cohort, for which two semesters of data were available. This focus allowed 
researchers to test the placement model for predictive fit whether the student opted for math 
in fall 2018 or deferred taking a math course until spring 2019. 


2 The analytical sample includes only records with complete data on the predictors and outcomes used in the study. There does 
not appear to be any bias associated with the missing data patterns. The distribution of students by placement category and 
by major (STEM/non-STEM) is the same for both the analytic sample and excluded sample. Although the sample population 
includes a larger percentage of Hispanic students and a smaller percentage of White students than does the population of the 
excluded sample, analytical tests conducted on the excluded sample show a similar pattern of relationship between available 
predictors and outcomes. 
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Table 1. Study sample for 2018 entering student cohort 


QR Placement Category Non-STEM Major STEM Major Ke) e-] | 
I 733 3,171 3,924 

ll 15,452 9,549 25,001 

ill 4,761 638 5,399 

IV 3,178 2,175 5,353 

Total 24,144 15,533 39,677 


Note: Total number of first-year entering students who attempted a math course in the 2018/19 academic year was 57,880. 
Sample was refined to include only students for whom complete information was available on all four variables of interest in 
this study (SAT-math score, EAP-math score, weighted high school math GPA, and weighted high school GPA). 


Table 2. Study sample for 2019 entering student cohort 


QR Placement Category Non-STEM Major STEM Major Ke) e-]| 
I 742 3,291 4,033 

ll 11,121 8,210 19,331 

ll 3,809 631 4,440 

IV 1,805 1,909 3,714 

Total 17,477 14,041 31,518 


Note: Total number of first-year entering students in 2019 who attempted a math course in the fall 2019 term was 44,503. 
The sample was refined to include only students for whom complete information was available on all four variables of 
interest in this study (SAT-math score, EAP-math score, weighted high school math GPA, and weighted high school GPA). 
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Methodology 


WestEd analysts conducted a series of logistic regression analyses to answer the research 
questions. The outcome of interest of each analysis is a binary variable indicating whether a 
student passed their first college math course taken, with passing defined as having earned a 
grade of C minus or higher.? For students who took more than one math course ina given 
semester, the course taken for the most credits was designated as the first math course. If 
courses had the same number of credits, WestEd analysts selected the course in which the 
student earned the higher grade. The predictors include SAT-math score, EAP-math score, 
weighted high school math GPA, and weighted high school GPA. According to the placement 
criteria, those predictors (along with other measures) are used in placing first-year college 
students into different categories. Analysts chose to use these four measures in the predictive 
model because they were the four measures that were available for the largest percentage of 
students. Additional information on the model is in Appendix A. 


To determine which model fits the data best, analysts looked at various statistics that include 
the log-likelihood function, Akaike information criterion (AIC), and Bayesian information 
criterion (BIC). A model with a smaller value of AIC or BIC is considered more accurate. Analysts 
also examined how well the model predicts and classifies the students by looking at the positive 
predicted value, classification accuracy, and the area under a Receiver Operating Characteristic 
(ROC) curve. The positive predicted value (in percentage) is the proportion of students who are 
predicted to pass their first college math course and actually do so, divided by the total number 
of students who are predicted to pass their first college math course. The classification 
accuracy is the proportion of students who are predicted to pass their first math course and 
actually do so, plus the number of students who are predicted not to pass their first math 
course but actually do so, divided by the total number of students in the analytic sample. ROC is 
a probability curve. The area under the curve represents the degree to which, or the measure 
of how much, the model can distinguish between students who can and cannot pass their first 
math course with a C minus or higher grade. It ranges between 0 and 1, with 0 being 
completely indistinguishable (the worst scenario) and 1 being 100 percent distinguishable (the 
perfect scenario). 


Consistent with the placement criteria, the analysis was conducted by category and by major 
(non-STEM versus STEM) for a given category. Students placed into Category | were not 
included in the study as they were deemed to have already met the General Education 
requirement in QR and therefore not required to take a math or quantitative reasoning course. 


3 In asmall number of cases, students earned credit for the first college math course with a “P” grade, and those instances are 
counted as successful completion. 
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How Multiple Measures Are Used for QR Placement Under 
EO 1110 


Although many different possible measures can be used to determine student placement in QR 
courses under EO 1110, the placement process uses a somewhat hierarchical approach.‘ Test 
scores are considered first, then a combination of test scores and high school senior year 
experience, and then high school grades. As a result, there are several ways in which a student 
can meet the criteria for being placed into Category | or Il, and although many students may 
meet several of these thresholds, the measures are considered in order. Therefore, placement 
into a category is made based on the first designated threshold that a student meets. 


Placing Students in Category II 


Box 2 provides an example to illustrate the decision tree that is used to determine how 
students who do not meet the criteria for Category | might be placed into Category Il. 


Box 2: Determining Placement in QR Category Il 


Students who do not meet the criteria for being placed in QR Category | can be designated as 
Category II (eligible for General Education QR without additional supports) if they meet at least 
one of the following measures, which are considered in order: 


e Math test score (on the SAT, ACT, or EAP) is at or above a designated threshold. 


e Math test score (on the SAT, ACT, or EAP) is below the designated Category II cutoff but 
above a slightly lower threshold and the student has completed a senior year math 
experience beyond algebra. 


e Overall high school GPA is 3.7 or greater. 


e For non-STEM majors, student has a combination of an overall GPA of at least 3.5 and 
completion of four years of math/quantitative reasoning or a math GPA of 3.0 or higher 
and completion of a senior year math experience? or fifth year of high school math. 


e For STEM majors, student has a combination of a math GPA of at least 3.5 and 
completion of a senior year math experience or fifth year of high school math. 


4 For a full list of options for placement in Categories | through IV for both QR and WC, see “CSU Placement of First Year 
Students Based on Academic Preparation,” https://calstate.policystat.com/policy/6656541/latest/. 


5 A senior year math experience is defined as a course taken during the senior year of high school that has either Algebra II or 
Integrated Math Ill as a prerequisite. 
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Because of the way the placement algorithm works, the highest percentage of students in the 
sample for this study were placed into Category II based on either their SAT-math scores, EAP- 
math scores, or performance on these tests in combination with having completed a senior 
year math experience (Table 3): 


e Almost 83 percent of students in STEM majors and 64 percent of non-STEM majors who 
were placed in Category II met that threshold based on measures that included either 
test scores alone or test scores in combination with math course taking in high school. 


e Amuch smaller percentage of students who were placed in Category Il were so placed 
based solely on having a high school GPA of 3.7 or higher (7.2 percent of STEM majors 
and 11.3 percent of non-STEM majors), but many of the students who were placed into 
the category based on a test score measure would also likely have met this GPA 
threshold. 


There are several measures that were used to place less than 3 percent of students into 
Category Il, potentially raising the question of the value of including those measures in the 
placement algorithm. Notably, a very small percentage (just over 1 percent) of STEM students 
in the sample who were placed into Category II were placed based solely on their math GPA. 
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Table 3. Percentage of Category II students in cohort by placement measure 


Percentage of Percentage of 
non-STEM STEM students 
students placed placed in 


Pl tM 1D) ipti 
Greeti alicia et eam ae in Category II by Category II by 


meeting this meeting this 
measure measure 


Placement based on SAT-math score; student has scored 570 or higher 22.9% 30.74% 
on the math section of the new SAT (effective fall 2016) or scored 550 or 
higher on SAT subject test in math (Level I or II) 


Placement based on initial EAP-math score 10.18% 21.22% 


Placement based on SAT-math result and successful completion of an 11.99% 11.82% 
approved senior experience; student was formerly conditionally placed 
based on SAT-math test result 


Placement based on EAP-math; student was formerly conditionally 19.44% 19.08% 
placed based on EAP-math score and successfully completed an 
approved senior experience 


Placement based on high school GPA 2 3.7 11.33% 7.23% 


Placement based on high school GPA 2 3.5 and 4 years of math (Area C 7.42% .89%* 
and Area G) — non-STEM 


Placement based on math GPA 2 3.0 and senior year math — non-STEM 6.65% 75% 


EAP-math “Standard Met: Conditionally Ready for CSU” and 4 years of 2.39% .22% 
math (Area C and Area G) — non-STEM 


Placement based on math GPA 2 3.5 and senior year math — STEM .14%* 1.25% 


Note: Sample includes 25,002 first-year students in QR Category II for whom information was available on each of four 
placement variables: SAT-math, EAP-math, overall GPA, and math GPA. Totals do not add to 100 percent. 
*In a small number of cases, placement codes intended only for non-STEM majors were given to STEM majors and vice versa. 


Placing Students in Category III or IV 


Although there are many different ways that a student may meet criteria for placement in 
Category Il, the determination of whether a student will be designated as Category III or IV is 
based solely on high school grades. Students who do not meet any of the criteria for placement 
in Category II but have a math GPA of 3.3 or better are placed into Category III regardless of 
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major; non-STEM majors can also be designated as Category III if their overall GPA is 3.0 or 
better. Those who do not meet any of these cutoffs are placed into Category IV. The analysis 
shows that approximately 98 percent of non-STEM students who were placed into Category III 
had an overall GPA of 3.0 or better. Even though STEM students are supposed to have a math 
GPA of 3.3 or higher in order to be placed into Category III, only about 48 percent of the 
Category III STEM students in the study’s sample were placed according to their math GPA. 
Others were placed based on their overall GPA, suggesting that not all campuses differentiated 
between STEM and non-STEM students for placement into Category Ill. 


Findings 


Model Fit (SAT-Math/EAP-Math/Math GPA/High School GPA) 


In most of the comparisons, a model that includes all four predictors fits the data better than 
models with fewer predictors. The high school GPA is the strongest predictor (with the most 
significant and higher odds ratio), followed by either the SAT-math score or the EAP-math score 
(both are the significant predictors). However, based on the model-fit statistics used in the 
study, a model without the high school math GPA (but one that does include the other three 
predictors) could perform equally well, compared with the model with all four predictors. 
Appendix B provides additional details on the correlations, sensitivity and predictive values for 
the models. 


Using High School GPA Alone to Predict Students’ Success in Their First 
College Math Course 


As indicated earlier, the high school GPA, out of any of the predictors used in the analyses, 
tended to be the strongest predictor of students’ success in their first college math course. In 
most cases, a model that includes high school GPA alone could predict the outcomes similarly 
to the model with all four predictors included (see Appendix B). To further explore how 
sensitive the model with high school GPA alone is in predicting students’ success in their first 
college math course, analysts ran additional regression analyses by category and by major 
within a given category. Because the placement criteria for Category Ill are similar to those for 
Category IV, the research team compared the findings between those two groups to 
understand how the students may perform similarly or differently. 


Category II Findings: High School GPA 


Non-STEM Students 

Based on the analysis of regressing the passing of the first college math course on high school 
GPA, a graph was plotted to demonstrate the relationship between the high school GPA and 
the predicted probability of passing the first college math course. Figure 1 indicates a positive 
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correlation: the higher the high school GPA, the higher the predicted probability that a first- 
year student would succeed in their first math course. For example, with a high school GPA 
equal to 3.0, the predicted probability to pass is about 70 percent; when GPA increases to 3.5 
or 3.7, the likelihood rises to about 83 percent or 87 percent, respectively. Based on the 
multiple measures placement criteria, non-STEM students who had a high school GPA of 3.7 or 
higher can be placed into Category II, regardless of their performance on other measures. The 
vertical red line on Figure 1 shows that cutoff mark. 


Figure 1. Predicted probability of passing the first college math course based on only 
high school GPA for non-STEM students placed in Category II 
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STEM Students 

The pattern for STEM students in Category II is quite similar to the one for the non-STEM 
students, though the corresponding predicted probability for a given high school GPA appears 
lower (Figure 2). For example, for STEM students with a high school GPA of 3.0, their predicted 
probability of passing their first math course is 67 percent; that probability increases to 79 
percent for those with a high school GPA of 3.5 and to 83 percent for those with a GPA of 3.7. 
The slightly lower predicted probability for passing the first math course for STEM versus non- 
STEM students may be due to the fact that STEM students are more likely enrolling in more 
intensive algebra-based courses, including calculus. 


Figure 2. Predicted probability of passing the first college math course based on high 
school GPA for STEM students placed in Category II 
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Category Ill and Category IV Findings: High School GPA 


The main difference between placement in Category III and Category IV is that students in 
Category IV are technically required to attend an Early Start program (though many choose not 
to do so),° whereas Early Start is recommended for students in Category III. Students in both 
categories are placed in entry-level math courses that provide some kind of additional support. 
Because students in both categories are often placed in the same supported courses on a given 


6 See Bracco et al. (2021b) for more information on Early Start under EO 1110. 
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campus, the outcomes for Categories Ill and IV are compared together in the following 
analyses. 


Non-STEM Students 

There was a similar pattern of predictive probability for non-STEM students placed in 
Categories Ill and IV (Figure 3). Although Category III students with high school GPAs below 3.0 
had a slightly higher chance of succeeding in their first college math course than their 
counterparts in Category IV, there is little difference in predictive probability for students with a 
GPA of 3.0 or higher. 


According to the placement criteria, those non-STEM students in Category III with a high school 
GPA below 3.0 would have been placed in Category IV unless their high school math GPA was 
3.3 or above. If students were placed into Category III based on their higher math GPA, it might 
not be surprising to see a slightly higher likelihood of passing that first college math course than 
their counterparts who did not do quite as well in their high school math courses. However, the 
overall curve for both categories is remarkably similar, raising the question of the value of 
distinguishing between these two placement categories. 


Figure 3. Predicted probability of passing the first college math course based on high 
school GPA for non-STEM students placed in Category Ill and Category IV 
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STEM Students 

Based on the placement criteria, the math GPA is the only criterion used to distinguish between 
placement in Category IIl versus Category IV for STEM students, regardless of overall high 
school GPA. However, this study’s analysis showed that a high percentage of STEM students 
who were placed into Category III actually had math GPAs below the 3.3 cutoff, suggesting that 
there was either a problem with the math GPA variable or, more likely, that some campuses did 
not differentiate between STEM and non-STEM majors when placing students in Category Ill. 
Those Category III students with a high school GPA equal to or less than 2.5 had a lower chance 
than their counterparts in Category IV of passing their first college math course (Figure 4), 
which implies that those students should have been placed in Category IV or that the current 
criterion to place STEM students in Category Ill may not work as expected. In other words, the 
use of high school math GPA alone to place STEM students in Category III may not be sufficient 
to predict future success in their first college math course. Adding another factor such as high 
school GPA, a strong predictor as indicated by this study’s analyses, may improve the 
classification accuracy and effectiveness in the placement decision. 


Figure 4. Predicted probability of passing the first college math course based on high 
school GPA for STEM students placed in Category III and Category IV 
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Discussion 


This study’s analyses illustrate that a student’s overall high school GPA is the best predictor of 
success in a first college math course for students in all placement categories in the CSU 
system. Because of the way the system’s placement algorithm works, a high percentage of 
students were actually placed into Category II based on their test scores, but the analyses 
suggest that the GPA is a better predictor of whether a student will be successful in that first 
course. The findings raise several questions for consideration with regard to the use of multiple 
measures for QR placement. 


What Are the Appropriate GPA Cutoffs for Placement into Category II? 


Currently, if a student does not meet one of the test score thresholds, the GPA cutoff is 3.7 for 
placement into Category II. Based on the study’s analysis, students with that GPA who are non- 
STEM majors have an 87 percent probability of passing their first college math course with a 
grade of C minus or better; for STEM majors the probability is 83 percent. A slightly lower GPA 
of 3.5 still yields a relatively high probability of passing their first college math course: 83 
percent for non-STEM majors and 79 percent for STEM majors. Interpreting the 
appropriateness of these likelihood estimates is a matter of making a policy choice regarding 
the question: What should the targeted predictive passing rate be to determine whether the 
GPA cutoff for Category II placement is appropriate or whether it should be lowered? 


What Is the Role of Standardized Testing, Now and in the Future? 


In general, the EO 1110 multiple measures policy is based on a hierarchical decision tree — 
students can be placed into Category II by meeting one of a number of different thresholds, 
whether through high school overall GPA, high school math GPA, standardized test scores, or a 
combination of these measures with completion of a senior year math experience. Based on 
the study’s analysis, adding the SAT and EAP scores to the model does very little to increase the 
predictive strength of the measures. For the next two years at least, because of COVID-19 
testing policies, these scores will not likely be available as students will not have had the 
opportunity to take these exams. Many institutions across the country are moving away from 
requiring standardized tests such as the ACT and SAT, and the CSU system could move in that 
direction as well. Given the uncertainty of whether such standardized tests will continue to be 
required for admission, the CSU system may wish to consider how or whether other measures 
might be added to the decision architecture. 


What Additional Measures Should Be Considered for Placement? 


Although a student’s math GPA might be expected to be a significant predictor of success in 
their first college math course, the analysis shows that this measure really does not add to the 
predictive strength of the model. The math GPA is so highly correlated with the overall GPA 
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that its effect is mostly washed out. In addition, very few students were placed using this 
measure, suggesting that it did not have a significant impact on placement for Category II. 


One factor that this study cannot fully address is the impact of a student’s high school senior 
year math experience in predicting success in their first college math course. Analyzing 
differences in senior year experiences and the extent to which those courses help predict 
success in first college math courses is beyond the scope of this report but may be an important 
measure for future consideration, particularly as it is the one measure that is used in 
combination with other measures (whether test scores or grades). 


Should There Be a Distinction Between Category III and Category IV? 


The multiple measures policy uses only grades to differentiate between Category Ill and 
Category IV placement for students. For those who are in non-STEM fields, either the overall 
high school GPA or high school math GPA can result in placement in Category III; for STEM 
majors, only the math GPA is used to differentiate between Categories Ill and lV. However, the 
study’s analyses show very similar outcomes for students in their first college math course 
based on their high school GPA, raising questions about whether there are significant enough 
differences in the support needs of the two categories to warrant the separation. Currently, the 
difference between placement in Category III versus Category IV is that students in Category IV 
are required to attend a summer Early Start program, whereas students in Category III are 
recommended to do so. However, the study team’s most recent report in this series on EO 1110 
illustrates that despite the requirement, not all students who are placed in Category IV enroll in 
Early Start (Bracco et al., 2021b),’” which further calls into question the value of distinguishing 
between these two placement categories. 


Additional Considerations 


Consideration of the above policy questions will be an important next step as the CSU system 
tries to determine how best to set up students for success in entry-level math courses. 
Additional analyses may be necessary to understand both the longer term implications of the 
placement policy for STEM students as well as the extent to which results of the analyses might 
look different on different campuses. Although understanding the extent to which these 
measures are good predictors of students’ success in their first college math course is 
important, that is only an initial step, particularly for students in STEM fields. For STEM 
students, understanding how successful they are as they progress through a STEM math 
sequence will be critical. 


7 In WestEd’s analysis of Early Start outcomes on seven CSU campuses, only about 57 percent of students placed in QR Category 
IV enrolled in Early Start, despite being technically required to do so. 
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Finally, while this study’s analyses can illustrate systemwide patterns, they do not take into 
account the nuances of the different courses and different supports provided on individual 
campuses. Understanding analytically the extent to which differences in high school GPA lead 
to different first-course outcomes on an individual campus, and whether those differences are 
greater for STEM versus non-STEM students, might be useful. In addition, understanding the 
relative success in first courses of students who have been placed in Categories III and IV on an 
individual campus will be important to understanding whether the types of support models 
currently provided need to be reexamined. 
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Appendix A 


Model Specification 


The study used a logistic regression to estimate the relationship between the following four 
predictors and a binary outcome of students’ passing their first college math course during their 
first year in college: the student’s scale score on the SAT’s math section (SAT-math), their scale 
score on the Early Assessment Program/Smarter Balanced Assessment (EAP-math), their 
weighted high school math grade point average (GPA), and their weighted high school overall 
GPA. 


The model takes the following form: 
Pr(pass = 1) = logit*(Bo + BiSAT; + B2EAP; + B3Math; + BaHS; + i) 


in which the subscript refers to student /, and in which SATj, EAP;, Mathi, and HS; represent the 
score or GPA of the corresponding predictor for student /. Bo and B1-B4 are parameters 
(coefficients) to be estimated from the data that are presented as odds ratios; each coefficient 
tells the reader how the odds of students’ passing their first college math course change for a 
one-unit change in the predictor. For example, B1 indicates how the odds of passing the first 
college math course vary with a one-unit change in the SAT-math scale score. gi represents the 
residual error term where gj ~ N(0, 9). The logit function was used because the outcome 
variable is binary. This model is described more fully by Tabachnick and Fidell (2018). 
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Appendix B 


Tables B.1 through B.6 provide the summary of model-fit statistics for students by category and 
STEM/non-STEM for the 2018 and 2019 cohorts. As noted in the body of the report, analysts 
looked at various statistics, including the log-likelihood function, Akaike information criterion 
(AIC), and Bayesian information criterion (BIC). A model with a smaller value of AIC or BIC is 
considered more accurate. The analysis also looked at how well the model predicts and 
classifies the students by looking at the positive predicted value, the classification accuracy, and 
the area under a Receiver Operating Characteristic (ROC) curve. The positive predicted value (in 
percentage) is the number of students who are predicted to pass their first college math course 
and actually do so, divided by the total number of students who are predicted to pass their first 
college math course. The classification accuracy is the number of students who are predicted to 
pass their first college math course and actually do so, plus the number of students who are 
predicted not to pass their first college math course but actually do so, divided by the total 
number of students in the analytic sample. ROC is a probability curve. The area under the curve 
represents the degree to which, or the measure of how much, the model can distinguish 
between students who can and cannot pass their first college math course with a C minus or 
higher grade. It ranges between 0 and 1, with 0 being completely indistinguishable (the worst 
scenario) and 1 being 100 percent distinguishable (the perfect scenario). 
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Table B.1. Category II, 2018 cohort: Summary of model-fit statistics along with classification information 


Predictor 


(ole fo KMe-}a(e) 


Koy :4 
eli arexeye | 


Positive 


ie Classification 
predictive 


accuracy (%) 


value (%) 


-6689.990 13389.98 | 13428.21 82.57 82.16 0.6982 | 15,452 
SAT-math 1.002764 <.001 
1 EAP-math (SS) 1.004344 <.001 
Math GPA 0.923426 0.097 
High school GPA | 4.348932 <.001 
-6691.372 13390.74 | 13421.33 82.55 82.18 0.6977 | 15,452 
> SAT-math 1.002789 <.001 
EAP-math (SS) 1.004274 <.001 
High school GPA | 4.063464 <.001 
3 High school GPA | 4.099403 <.001 -6917.99 13839.98 |13855.27 82.21 82.17 0.6493 | 15,452 
Category II: STEM 
-4508.719 9027.438 | 9063.259 80.01 79.64 0.6826 9,549 
SAT-math 1.002031 <.001 
1 EAP-math (SS) 1.004494 <.001 
Math GPA 1.040388 0.475 
High school GPA | 3.243993 <.001 
-4508.974 9025.947 | 9054.604 80.01 79.62 0.6825 9,549 
2 SAT-math 1.002032 <.001 
EAP-math (SS) 1.004539 <.001 
High school GPA | 3.363619 <.001 
3 High school GPA 3.672828 <.001 -4634.834 9273.668 | 9287.997 LOI2 79.67 0.6400 9,549 
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Table B.2. Category III, 2018 cohort: Summary of model-fit statistics along with classification information 


Positive Bash tte 
| oy4 Classification 


Predictor Coyote LMe-hate) likelihood | predictive accuracy (%) 


value (%) 


-2740.277 5 5490.554 | 5522.896 72,23 71.81 0.6270 | 4,761 


SAT-math 1.003382 <.001 
1 EAP-math (SS) 1.004359 <.001 
Math GPA 0.986119 0.852 
High school GPA 2.162869 <.001 
-2740.295 4 5488.589 | 5514.462 72.23 71.81 0.6270 | 4,761 
9 SAT-math 1.003381 <.001 
EAP-math (SS) 1.004351 <.001 
High school GPA 2.145840 <.001 
3 High school GPA 2.629556 <.001 -2822.918 2 5649.835 | 5662.772 71.59 71.56 0.5564 | 4,761 
Category III: STEM 
-352.939 5 715.8783 738.17 74.20 73.67 0.6291 638 
SAT-math 1.002932 0.173 
al EAP-math (SS) 1.001912 0.264 
Math GPA 0.618700 0.012 
High school GPA 4.812405 <.001 
-356.1263 3 718.2527 | 731.6277 74.13 73.67 0.6170 638 
2 Math GPA 0.676112 0.034 
High school GPA 5.244016 <.001 
3 High school GPA 4.154648 <.001 -358.4219 2 720.8439 | 729.7606 74.17 74.14 0.6008 638 


=24= 


Using Multiple Measures to Predict Success in WWestEd 2. 
WestEd.org 


Students’ First College Math Course 


Table B.3. Category IV, 2018 cohort: Summary of model-fit statistics along with classification information 


Positive pees 
Keoy4 Classification 


Teetellacelg Odds ratio likelihood predictive accuracy (96) 


value (%) 


-2062.796 5 4135.592 | 4165.912 63.22 62.02 0.6236 | 3,178 
SAT-math 1.004173 <.001 
1 EAP-math (SS) 1.002585 <.001 
Math GPA 1.156338 0.133 
High school GPA 2.934581 <.001 
-2063.923 4 4135.845 | 4160.101 63.14 61.93 0.6233 | 3,178 
2 SAT-math 1.004166 <.001 
EAP-math (SS) 1.002661 <.001 
High school GPA 3.160068 <.001 
3 High school GPA 3.149128 <.001 -2108.183 2 4220.365 | 4232.493 61.16 61.11 0.5711 | 3,178 
Category IV: STEM 
-1353.713 5 2717.426 2745.850 67.25. 65.61 0.6524 | 2,175 
SAT-math 1.004106 <.001 
1 EAP-math (SS) 1.004168 <.001 
Math GPA 0.840039 0.160 
High school GPA 1.004106 <.001 
-1354.703 4 2717.407 2740.146 66.83 65.06 0.6515. | 2,175 
SAT-math 1.004084 <.001 
: EAP-math (SS) 1.004028 <.001 
High school GPA 2.808969 <.001 
3 High school GPA 2.963908 <.001 -1397.976 2 21799,952 2811.322 64.35 63.45 0.6004 | 2,175 
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Log Positive | Classification 
Predictor foto feKWe-h aie) p-value likelihood predictive mecuraey (el 
value (%) 
| Category IV: All Majors 
-3429.262 5 6868.523 | 6901.45 64.75 63.40 0.6301 | 5,353 
SAT-math 1.004005 <.001 
1 EAP-math (SS) 1.002965 <.001 
Math GPA 0.996882 0.967 
High school GPA 2.475579 <.001 
-3429.262 4 6866.525 | 6892.867 64.76 63.42 0.6301 | 5,353 
SAT-math 1.004005 <.001 
2 EAP-math (SS) 1.002963 <.001 
High school GPA 2.471153 <.001 
3 High school GPA 2.752239 <.001 -3508.964 2 7021.928 | 7035.099 62.33 62.08 0.5835 | 5,353 
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Table B.4. Category II, 2019 cohort: Summary of model-fit statistics along with classification information 


oy 4 EMS Classification 
likelihood DF predictive AUROC 


WW) Foye (| Predictor (ole lok Me-ha(e) 
accuracy (%) 


value (%) 


-4827.501 5 9665.001 | 9701.584 82.36 81.86 0.7074 | 11,121 
SAT-math 1.003380 | <.001 
1 EAP-math (SS) 1.004280 | <.001 
Math GPA 0.959387 | 0.500 
High school GPA | 4.634492 | <.001 
-4827.729 4 9663.458 | 9692.724 82.34 81.86 0.7073 | 11,121 
9 SAT-math 1.003401 | <.001 
EAP-math (SS) 1.004241 | <.001 
High school GPA | 4.482988 | <.001 
3 High school GPA | 4.460478 | <.001 -5021.364 2 10046.73 | 10061.36 81.86 81.72 0.6631 | 11,121 
Category II: STEM 
-3963.043 2 7936.087 7971.752 73S 78.77 0.6794 8,210 
SAT-math 1.003966 | <.001 
1 EAP-math (SS) 1.003768 | <.001 
Math GPA 0.968526 | 0.608 
High school GPA | 2.914891 | <.001 
-3963.175 4 7934.350 7962.402 79:25 78.81 0.6793 8,210 
2 SAT-math 1.003962 | <.001 
EAP-math (SS) 1.003732 | <.001 
High school GPA | 2.830442 | <.001 
3 High school GPA 2.995640 | <.001 -4114.949 2 8233.899 8247.925 78.82 78.76 0.6208 8,210 
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Table B.5. Category III, 2019 cohort: Summary of model-fit statistics along with classification information 


Positive oe 
| Koy:4 : Classification 


HEI! likelihood Predicive accuracy (%) 


value (%) 


-2184.580 5 4379.161 | 4410.386 72.24 71.83 0.6346 | 3,809 
SAT-math 1.004104 <.001 
1 EAP-math (SS) 1.003540 <.001 
Math GPA 0.939989 0.511 
High school GPA | 2.990771 <.001 
-2184.796 4 4377.593 | 4402.573 72.28 71.88 0.6342 | 3,809 
> SAT-math 1.004099 <.001 
EAP-math (SS) 1.003492 <.001 
High school GPA | 2.883322 <.001 
3 High school GPA | 3.694222 <.001 | -2244.02 2 4492.040 | 4504.530 71.66 71,59 0.5771 | 3,809 
Category III: STEM® 
-351.407 5 712.813 | 735.0495 74.36 72.90 0.7001 631 
SAT-math 1.006057 0.004 
1 EAP-math (SS) 1.007135 <.001 
Math GPA 0.618559 0.010 
High school GPA | 7.548898 <.001 
3 High school GPA | 5.684423 <.001 -376.4501 2 756.900 | 765.795 70.70 70.52 0.6064 | 631 


8 All four predictors were significant in Model 1 for Cat Ill STEM students; with no non-significant predictors to eliminate from the model, Model 2 was not run for this subgroup. 
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Table B.6. Category IV, 2019 cohort: Summary of model-fit statistics along with classification information 


Predictor 


| oy:4 
likelihood 


Positive 
predictive 


value (%) 


Classification 
accuracy (%) 


-1166.349 2342.697 | 2370.189 63.65 62.83 0.6111 | 1,805 
SAT-math 1.002812 0.011 
1 EAP-math (SS) 1.003062 <.001 
Math GPA 1.056999 0.676 
High school GPA 3.086860 <.001 
-1166.436 2340.872 | 2362.865 63.67 62.88 0.6108 | 1,805 
2 SAT-math 1.002825 0.011 
EAP-math (SS) 1.003097 <.001 
High school GPA 3.171694 <.001 
3 High school GPA 3.174266 <.001 | -1188.058 2380.116 | 2391.113 62.62 62.77 0.5640 | 1,805 
Category IV: STEM 
-1150.988 2311:977 | 2339:749 69.55 67.57 0.6863 | 1,909 
SAT-math 1.006538 <.001 
1 EAP-math (SS) 1.004799 | <,001 
Math GPA 1.087091 0.541 
High school GPA 3.159485 <.001 
-1151.175 2310.35 2332.567 69.59 67.68 0.6862 | 1,909 
SAT-math 1.006548 <.001 
2 EAP-math (SS) 1.004847 | <.001 
High school GPA 3.307686 <.001 
3 High school GPA 3.661321 <.001 | -1215.662 2435.325 | 2446.433 64.85 63.96 0.6199 | 1,909 
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ote x O O a ae A O AURO 
‘Category IV: All Majors 
-2340.203 4690.407 | 4721.506 66.02 64.54 0.6419 | 3,714 
SAT-math 1.004281 <.001 
1 EAP-math (SS) 1.003595 <.001 
Math GPA 1.026627 0.780 
High school GPA | 2.304588 <.001 
-2340.242 4688.485 | 4713.364 66.00 64.51 0.6420 | 3,714 
SAT-math 1.004287 <.001 
: EAP-math (SS) 1.003611 <.001 
High school GPA | 2.339051 <.001 
3 High school GPA | 2.758112 <.001 | -2410.944 4825.888 | 4838.328 63.27 63.09 0.5879 | 3,714 
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